Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adds a key-value endpoint to store the state of dashboard filters #17536

Merged

Conversation

michael-s-molina
Copy link
Member

@michael-s-molina michael-s-molina commented Nov 24, 2021

SUMMARY

Currently, the filters state is being stored in the URL to allow users to preserve their current interactions in case of a refresh and also to share specific states with other users. The most common example is when a user, after exploring a dashboard using filters, wishes to share their findings with other users.

The problem with this approach is that when the size of the state is big, it provokes URL length problems with the major browsers.

The proposed solution in this PR is to store the filters state on the server-side and keep in the URL only a secure key that gives access to the stored content. To do that, we'll extend the current dashboard endpoint with an additional path that supports the storage of information in the form of key-value entries. By making this endpoint relative to the main resource we can reuse the security model and access grants.

The additional endpoint and the supported operations are:

POST api/v1/dashboard/<id>/filter_state

BODY 
{
   // Any type of JSON supported text.
   value: <string>,
}

RESPONSE
{
   // The key generated on the server-side
   key: <string>
}
PUT api/v1/dashboard/<id>/filter_state/<key>

BODY
{
   // Any type of JSON supported text.
   value: <string>,
}

RESPONSE
{
   // The result of the operation
   message: <string>
}
GET api/v1/dashboard/<id>/filter_state/<key>

RESPONSE
{
   // Any type of JSON supported text.
   value: <string>,
}
DELETE api/v1/dashboard/<id>/filter_state/<key>

RESPONSE
{
   // The result of the operation
   message: <string>
}

The keys are generated on the server-side using the https://docs.python.org/3/library/secrets.html#secrets.token_urlsafe function and the value can be any JSON supported text.

To store the key-value entries, this PR uses Flask-Caching, which enables support to different backends like Redis, Memcached, file system, etc. It also allows the configuration of expiration times for the stored entries in order to discard the ones that are not being used anymore and optimize the required storage space.

This PR also allows a configuration to preserve actively accessed entries by renewing the expiration time every time a value is retrieved.

Follow-up of #17337

TESTING INSTRUCTIONS

1 - Execute Python tests
2 - All tests should pass

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Nov 24, 2021

Codecov Report

Merging #17536 (b3841eb) into master (a959e6d) will decrease coverage by 0.29%.
The diff coverage is n/a.

❗ Current head b3841eb differs from pull request most recent head b0e5a42. Consider uploading reports for the commit b0e5a42 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master   #17536      +/-   ##
==========================================
- Coverage   77.01%   76.71%   -0.30%     
==========================================
  Files        1049     1062      +13     
  Lines       56682    56997     +315     
  Branches     7851     7851              
==========================================
+ Hits        43651    43723      +72     
- Misses      12778    13021     +243     
  Partials      253      253              
Flag Coverage Δ
hive ?
mysql 81.80% <ø> (-0.23%) ⬇️
postgres 81.81% <ø> (-0.23%) ⬇️
presto ?
python 81.89% <ø> (-0.64%) ⬇️
sqlite 81.49% <ø> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engines/hive.py 0.00% <0.00%> (-85.19%) ⬇️
superset/db_engine_specs/hive.py 70.27% <0.00%> (-16.99%) ⬇️
superset/db_engine_specs/presto.py 83.50% <0.00%> (-6.48%) ⬇️
superset/utils/cache_manager.py 96.15% <0.00%> (-3.85%) ⬇️
superset/views/database/mixins.py 81.03% <0.00%> (-1.73%) ⬇️
superset/connectors/sqla/models.py 86.85% <0.00%> (-1.58%) ⬇️
superset/models/core.py 89.26% <0.00%> (-0.74%) ⬇️
superset/connectors/base/models.py 88.19% <0.00%> (-0.49%) ⬇️
superset/db_engine_specs/base.py 88.15% <0.00%> (-0.39%) ⬇️
superset/charts/commands/export.py 94.11% <0.00%> (-0.17%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a959e6d...b0e5a42. Read the comment docs.

Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

Can you add some tests for PUT and DELETE as well?

superset/key_value/utils.py Outdated Show resolved Hide resolved
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick first pass, looking really good! Will review/test more tomorrow!

superset/config.py Outdated Show resolved Hide resolved
superset/key_value/api.py Outdated Show resolved Hide resolved
superset/key_value/schemas.py Outdated Show resolved Hide resolved
superset/key_value/utils.py Outdated Show resolved Hide resolved
@betodealmeida betodealmeida added the risk:db-migration PRs that require a DB migration label Nov 24, 2021
Copy link
Member

@craig-rueda craig-rueda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments around moving logic out of DAO layer and into the CMDs. Would also be great to see a few more tests, especially around the CMDs themselves.

superset/dashboards/filters_state/api.py Outdated Show resolved Hide resolved
superset/dashboards/filters_state/dao.py Outdated Show resolved Hide resolved
superset/dashboards/filters_state/dao.py Outdated Show resolved Hide resolved
@michael-s-molina
Copy link
Member Author

@betodealmeida @villebro @craig-rueda Tackled all your comments. Can you review it again? Thanks

Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! I left a few small comments, but it looks good to go, IMHO.

superset/config.py Outdated Show resolved Hide resolved
superset/dashboards/filter_state/api.py Show resolved Hide resolved
superset/key_value/commands/create.py Outdated Show resolved Hide resolved
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments - tested, and works really nicely, really looking forward to this!

superset/config.py Outdated Show resolved Hide resolved
superset/dashboards/filter_state/api.py Show resolved Hide resolved
superset/key_value/api.py Outdated Show resolved Hide resolved
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

from tests.integration_tests.fixtures.world_bank_dashboard import (
load_world_bank_dashboard_with_slices,
)
from tests.integration_tests.test_app import app
Copy link
Contributor

@mayurnewase mayurnewase Dec 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

app is reimported here.
mypy hook should have given error in ci too as my local does.
not sure how it was skipped.(different version or ignore config or mypy doesn't run in pipeline ?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out @mayurnewase. I submitted a fix in #17887. @villebro Do you have any idea how pre-commit and CI missed that?

@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels risk:db-migration PRs that require a DB migration size/XXL 🚢 1.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants